Earphones for measuring and entraining respiration

ABSTRACT

An earphone includes a loudspeaker, a microphone, a housing supporting the loudspeaker and microphone, and ear tip surrounding the housing and configured to acoustically couple both the loudspeaker and the microphone to an ear canal of a user, and to acoustically close the entrance to the user&#39;s ear canal. A processor provides output audio signals to the loudspeaker, receives input audio signals from the microphone, extracts a rate of respiration from the input audio signals, adjusts the output audio signals based on the extracted rate of respiration, and provides the adjusted output audio signals to the loudspeaker.

RELATED APPLICATIONS

This application is a division of U.S. patent application Ser. No.15/655,845, filed Jul. 20, 2017, which is related to, and incorporatesby reference, U.S. patent application Ser. No. 15/106,989, filed Jun.21, 2016; application Ser. No. 15/348,400, filed Nov. 10, 2016; andapplication Ser. No. 15/352,034, filed Nov. 17, 2016, all titledIntelligent Earplug System. It is also related to U.S. patentapplication Ser. No. 15/267,567, entitled Sleep Assistance Device;application Ser. No. 15/267,464, entitled Sleep Quality Scoring andImprovement; application Ser. No. 15/267,552, entitled IntelligentWake-Up System; application Ser. No. 15/267,848, entitled Sleep System;application Ser. No. 15/267,858, entitled User Interface for a SleepSystem; and application Ser. No. 15/267,886, entitled Sleep AssessmentUsing a Home Sleep System, all of which were filed on Sep. 16, 2016.This application is also related to U.S. patent application Ser. No.15/655,863, titled Sleep Assistance Device For Multiple Users, filedJul. 20, 2017, which is incorporated here by reference.

BACKGROUND

This disclosure relates to earphones for measuring and entrainingrespiration.

Sleeplessness and poor or interrupted sleep may significantly affect aperson's health. Poor sleep may be caused by such factors as ambientnoise, stress, medical conditions, or discomfort. Thus, there exists aneed for a sleep aid that can help address the underlying causes of poorsleep without adversely affecting the user's health in other, unintendedways.

SUMMARY

In general, in one aspect, a system includes an earphone, which includesa loudspeaker, a microphone, a housing supporting the loudspeaker andmicrophone, and ear tip surrounding the housing and configured toacoustically couple both the loudspeaker and the microphone to an earcanal of a user, and to acoustically close the entrance to the user'sear canal. A processor provides output audio signals to the loudspeaker,receives input audio signals from the microphone, extracts a rate ofrespiration from the input audio signals, adjusts the output audiosignals based on the extracted rate of respiration, and provides theadjusted output audio signals to the loudspeaker.

Implementations may include one or more of the following, in anycombination. Adjusting the output audio signals may include adjusting arhythm of the output audio signals to be about one cycle per minute lessthan the detected respiration rate. Adjusting the output audio signalsmay include transitioning the output audio signals from respirationentrainment sounds to masking sounds. Adjusting the output audio signalsmay include transitioning the output audio signals from masking soundsto awakening sounds. The earphone may include a memory storing soundfiles, and providing the output audio signals may include retrieving afirst sound file from the memory. Adjusting the output audio signals mayinclude retrieving a second sound file from the memory and using thesecond sound file to generate the output audio signal. The processor maybe integrated within the earphone. The processor may be integratedwithin a portable computing device.

The processor may extract the rate of respiration by detecting peakshaving a frequency of around 1 Hz in the input audio signals, based onthe detected peaks, computing an instantaneous heart rate, measuring afrequency of an oscillation within the instantaneous heart rate, andbased on the frequency of the oscillation, computing the rate ofrespiration. The processor may measure the frequency of the oscillationwithin the instantaneous heart rate by computing a fast Fouriertransform (FFT) of the instantaneous heart rate. The processor maymeasure the frequency of the oscillation within the instantaneous heartrate by computing a gradient of the instantaneous heart rate, andcomputing a fast Fourier transform (FFT) of the gradient of theinstantaneous heart rate. The processor may measure the frequency of theoscillation within the instantaneous heart rate by detecting peaks ofthe instantaneous heart rate. The processor may measure the frequency ofthe oscillation within the instantaneous heart rate by fitting a sinefunction to the instantaneous heart rate, the frequency of the sinecurve being the frequency of the oscillation. The system may include asecond earphone including a second loudspeaker, a second microphone, asecond housing supporting the second loudspeaker and second microphone,and a second ear tip surrounding the second housing and configured toacoustically couple both the second loudspeaker and the secondmicrophone to a second ear canal of the user, and to acoustically closethe entrance to the user's second ear canal, in which case the processorreceives second input audio signals from the microphone, and detects thepeaks having a frequency of around 1 Hz by combining the input audiosignals from the first microphone with the second input audio signals,and detecting peaks within the result of the combination. Combining theinput audio signals may include multiplying the amplitude of the firstinput audio signals by the amplitude of the second input audio signal,at each time that the two signals are sampled.

Providing the output audio signals to the loudspeaker may includeproviding signals which represent sounds across a first frequency band,the audio signals including a notch in which the sounds lack energywithin a second frequency band narrower than the first frequency band,and the processor may be configured to extract the rate of respirationby applying a band-pass filter to the input audio signals to limit theinput audio signals to a third frequency band contained within thesecond frequency band, and demodulating the filtered input audio signalsto compute a rate of respiration corresponding to energy in the inputaudio signals in the third frequency band. The third frequency band maybe coextensive with the second frequency band. The first frequency bandmay extend at least 40 Hz below a lower end of the second frequencyband. The second frequency band may extend between about 250 to 350 Hz.The earphone may include a memory storing sound files, and providing theoutput audio signals may include retrieving a first sound file from thememory, the first sound file representing audio signals corresponding tosounds having energy in the second frequency band, and providing theoutput audio signals includes applying a notch filter to audio signalsgenerated from the first sound file, to remove energy from the signalswithin the second frequency band. The first sound file may representaudio signals corresponding to sounds lacking energy in the secondfrequency band.

Advantages include acoustically sensing the respiration rate at the earwithout interference from audio signals being generated by the earphone.

All examples and features mentioned above can be combined in anytechnically possible way. Other features and advantages will be apparentfrom the description and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1 and 2 show cross-sectional views of earphones with an integratedmicrophones.

FIG. 3 shows an external view of the system of FIG. 1 or 2 .

FIGS. 4, 5 a, 5 b, and 5 c show audio spectrographs.

FIGS. 6 and 7 show graphs of data derived from the type of data shown inFIGS. 5 a -5 c.

FIGS. 8-10 show graphs of sensor readings

DESCRIPTION

Several of the above-referenced applications describe a bedside systemthat detects a user's respiration rate and uses that to infer and managetheir sleep state. In particular, to assist the user with fallingasleep, the system plays sounds that have a rhythm slightly slower thanthe user's own respiration rate. This naturally leads the user to slowtheir breathing to match the rhythm of the sounds, in a process referredto as entrainment. As the user slows their rate of respiration, the rateof the sounds is further reduced, in a feedback loop that leads the usergradually to sleep. Once the user falls asleep (as indicated byartifacts in their respiration rate), the system switches to playingmasking sounds, which diminish the user's ability to detect, and bedisturbed by, external sounds. If the user is detected to be waking uptoo early, entrainment may be reactivated. When it is time for the userto wake up, the system may coordinate wake-up sounds with the user'ssleep state and other information to wake the user in theleast-disruptive way possible.

Others of the above-referenced applications describe intelligentearplugs which the user can wear while sleeping, and which providemasking sounds through the night, and alarm or alert sounds when needed.These earplugs are controlled by a smartphone, but principally operateautonomously, playing stored masking sounds until instructed otherwiseby the controlling phone, or based on an internal clock. It would beadvantageous if the intelligent earplugs could play therespiration-entraining sounds of the bedside systems, to help the userfall asleep without disturbing others who may be sharing the bed orroom. One solution to that, described in U.S. application Ser. No.15/655,836, now U.S. Pat. No. 10,478,590, is for the sleep system toinform the earplugs of the user's respiration rate and sleep state, andfor the earplugs to adjust the rate of a rhythmic component in storedentrainment sounds as in the out-loud system.

This disclosure describes how to add respiration sensing to the earplugsthemselves, so that the external system is not required, and theearplugs can operate fully autonomously, or with only a smart phone tocontrol them.

As shown in FIGS. 1, 2 and 3 , sleep-sensing earphones 100 or 200include an ear tip sealing structure 102 that blocks, or occludes, theentrance to the ear canal. FIGS. 1 and 2 show cross-sections of twodifferent earphone examples, while FIG. 3 shows an exterior view, whichis the same for the examples of either FIG. 1 or 2 , for reference. Aretaining structure 104 helps retain the earphone in the ear, and putspressure on the sealing structure 102 to maintain the seal by pushing onthe concha, opposite to where the sealing structure meets the ear canal.The sealing structure 102 helps to passively block outside sounds fromentering the ear, increasing the effectiveness of the masking soundsplayed by the earphones.

Another result of occluding the ear canal is that sounds produced by thebody, such as the heartbeat and respiration sounds, are amplified withinthe ear canal. With the addition of a microphone 106 (FIG. 1 ) or 206(FIG. 2 ), the heartbeat can be sensed and its rate determined. Theprocessor 108 on-board each earphone (or in one, if they coordinatetheir action) can then extract the respiration rate from the heartbeatsignal, and adjust the timing of entrainment sounds being played to theuser through a speaker 110. In the example of FIG. 1 , the microphone106 and speaker 110 are shown behind a screen 112, as described in U.S.Pat. No. 9,635,452, which is incorporated here by reference. Themicrophone may be mounted near or on the speaker 110, or integrated intothe speaker housing 114. In the example of FIG. 2 , the microphone 206is mounted directly to the PCB 208 and the screen 212 is flat, or maynot be needed; the volume inside the earbud is coupled to the ear canalvia space around the speaker 110. As long as the earbud/ear canal systemis effectively sealed at the frequencies of interest, the microphonewill detect the targeted sounds coming from inside the ear canal. Otherconfigurations that couple the microphone acoustically to the ear canalwill also work.

A difficulty arises in attempting to use a microphone coupled to the earcanal to detect respiration while the earphones are simultaneouslyplaying sounds (and in particular, sounds which may not be significantlydifferent from the sound of breathing). One solution, as shown in FIG. 4, is to notch out a small frequency band of the entrainment or maskingsound, and to filter the microphone signal, shown in FIGS. 5 a-5 c fordifferent respiration rates, with a corresponding band-pass filter. Dueto the psychoacoustic phenomenon known as the upward spread offrequency, a user will not be able to audibly detect the small notch inthe entrainment or masking sound, but enough of the sound of theirrespiration will be detectible within the notched and filtered window tomeasure their respiration rate.

In particular, a notch in a range around 250-350 Hz will leave enoughenergy below the notch for the upper spread of frequency to hide thenotch from the user. More specifically, a notch between 260-340 Hz hasbeen found to be sufficient. The notch can either be removed from themasking or entrainment sound by a DSP during operation of the earplugs,or the stored sounds can simply have the notch already present Aband-pass filter matching, or narrower than, the notch band is thenapplied to the microphone signal (dashed lines 502, 504 in FIGS. 5 a-5 c), which can be visualized as energy over time, as shown by the solidline 522 in FIG. 6 . The respiration envelope is fit to the data, dashedline 524. A peak detection algorithm is applied, as shown in FIG. 7 , todetect the respiration of the user, the rate of the clusters 526 ofpeaks 528 corresponding to breaths per minute.

The human heartbeat is infrasonic, while acoustic signatures fromrespiration can be observed in the 100 s of Hz, so the heartbeat will betoo low-frequency (and the high-frequency part of the heartbeat impulsetoo low-energy) to interfere with detection of respiration in thenotched band. The heartbeat could also be removed from the microphonesignal using an additional heart rate sensor, such as aphoto-plethysmograph (PPG) sensor included in the earphones.

Alternatively, the heartbeat itself can be derived from the microphonesignals, and the respiration rate can be extracted from the heart ratevariability. Specifically, as shown in FIG. 6 , the microphone coupledto the occluded ear canal detects heartbeats as energy peaks in a signalwith a frequency of around 8-10 Hz (the heart rate itself is around 1Hz). As this rate is far below the frequency range of the maskingsounds, those sounds will not interfere with detecting the heartbeat. Ifboth ears are equipped with microphones, and the signals are transmittedto the smart phone (or from one ear to the other) for analysis,combining the amplitudes of the two signals at each time sample, such asby multiplication, can greatly increase the signal to noise ratio, asshown in FIG. 7 . Applying a peak-finding algorithm to the microphonesignal and observing the distance between consecutive peaks yields thebeat-to-beat or instantaneous, heart rate value, shown in FIG. 8 .

FIG. 8 shows that there is a cyclic variability to the instantaneousheart rate. The period of this variability happens to be the respirationrate—as the user inhales, their heart rate increases, and as theyexhale, their heart rate decreases. Applying another peak detectionstep, or other frequency analysis such as a fast Fourier transform (FFT)or fitting a sine function to the curve, to the instantaneous heart rateor to its gradient reveals the respiration rate.

If the earphones happen to include a feedback-based active noisereduction (ANR) system, to further block environmental sounds, thesystem microphone of the ANR system would be more than adequate fordetecting the sound of respiration or blood flow and measuring therespiration or heart rate, but it would be done within the feedbackloop, so notching the anti-noise output of the ANR system would not benecessary. However, an ANR system is likely to consume a lot of power,and may not be suitable or necessary for sleep-focused earphones. Sincethe respiration or heart rate sensing is very narrow-band, a simplerMEMS microphone should be sufficient and a much lower-power componentmay be used, benefiting the overall battery life and component size ofthe earphones. Similarly, it may be possible to use an external device,such as a smartphone, to filter and demodulate the microphone signals todetect the respiration rate or heart rate, and to modify the outputsounds accordingly, but battery life may be better served by doing allthe processing within the earphones. The trade-off between power forprocessing and power for communication may depend on factors unrelatedto the acoustics, including battery size, antenna placement, and memoryrequirements, to name a few.

Embodiments of the systems and methods described above comprise computercomponents and computer-implemented steps that will be apparent to thoseskilled in the art. For example, it should be understood by one of skillin the art that the computer-implemented steps may be stored ascomputer-executable instructions on a computer-readable medium such as,for example, hard disks, optical disks, solid-state disks, flash ROMS,nonvolatile ROM, and RAM. Furthermore, it should be understood by one ofskill in the art that the computer-executable instructions may beexecuted on a variety of processors such as, for example,microprocessors, digital signal processors, and gate arrays. For ease ofexposition, not every step or element of the systems and methodsdescribed above is described herein as part of a computer system, butthose skilled in the art will recognize that each step or element mayhave a corresponding computer system or software component. Suchcomputer system and software components are therefore enabled bydescribing their corresponding steps or elements (that is, theirfunctionality), and are within the scope of the disclosure.

A number of implementations have been described. Nevertheless, it willbe understood that additional modifications may be made withoutdeparting from the scope of the inventive concepts described herein,and, accordingly, other embodiments are within the scope of thefollowing claims.

What is claimed is:
 1. A method of adjusting sounds heard by a user ofan earphone, the method comprising: providing output audio signals to aloudspeaker supported by a housing and acoustically coupled to theuser's ear canal by an ear tip surrounding the housing and acousticallyclosing the entrance to the user's ear canal; receiving input audiosignals from a microphone in the housing and also acoustically coupledto the user's ear canal by the ear tip; and in a processor extracting arate of respiration from the input audio signals; adjusting the outputaudio signals based on the extracted rate of respiration; and providingthe adjusted output audio signals to the loudspeaker, wherein: the stepof providing the output audio signals to the loudspeaker comprisesproviding signals which represent sounds across a first frequency band,the audio signals including a notch in which the sounds lack energywithin a second frequency band narrower than the first frequency band;and the processor is configured to extract the rate of respiration byapplying a band-pass filter to the input audio signals to limit theinput audio signals to a third frequency band contained within thesecond frequency band; and demodulating the filtered input audio signalsto compute the rate of respiration corresponding to energy in the inputaudio signals in the third frequency band.
 2. The method of claim 1,wherein the step of adjusting the output audio signals comprisesadjusting a rhythm of the output audio signals to be about one cycle perminute less than the extracted rate of respiration.
 3. The method ofclaim 1, wherein the step of adjusting the output audio signalscomprises transitioning the output audio signals from respirationentrainment sounds to masking sounds.
 4. The method of claim 1, whereinthe step of adjusting the output audio signals comprises transitioningthe output audio signals from masking sounds to awakening sounds.